Method and apparatus for frame coding in vertical raster scan order for hevc

ABSTRACT

A method and apparatus for frame coding in adaptive raster scan order. The method includes encoding at least one of image or video utilizing input frames and at least one of a data related to the input frame to produce bitstream with raster scan order information and displacement information for producing compressed video bitstream, at decoding time, decoding at least one of the encoded bitstream with raster scan order information and displacement information for producing compressed video bitstream.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.14/561,816, filed Dec. 5, 2014, which is a continuation of U.S. patentapplication Ser. No. 13/250,806, filed Sep. 30, 2011, now U.S. Pat. No.8,934,729, which claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/388,478 filed Sep. 30, 2010, and which claimsthe benefit of U.S. Provisional Patent Application Ser. No. 61/474,435,filed Apr. 12, 2011, all of which are incorporated herein by referencein their entirety.

BACKGROUND OF THE INVENTION

Field of the Invention

Embodiments of the present invention generally relate to a method andapparatus for frame coding in vertical raster scan for HEVC.

Description of the Related Art

HEVC is a video coding standard being standardized to improve codingefficiency by 50% over H264/AVC. To achieve this goal, lots of newcoding tools have been proposed for HEVC. However, all the proposedcoding tools assume the horizontal raster scan of micro blocks, whichhas two problems: 1) Horizontal raster scan is not always effective inprediction especially for intra coding and 2) with sliding window memoryfor motion estimation (ME) and motion compensation (MC), large on-chipmemory is required to compensate for large vertical motion. This isespecially a challenging for UHD (ultra high definition) videos, suchas, 4k×2k and 8k×4k. While UHD video coding is one of the applicationareas of HEVC, the frame coding in horizontal scan order is not costeffective at all for UHD videos with high vertical motion.

Therefore, there is a need for a method and/or apparatus for framecoding for HEVC.

SUMMARY OF THE INVENTION

Embodiments of the present invention relate to a method and apparatusfor frame coding in adaptive raster scan order. The method includesencoding at least one of image or video utilizing input frames and atleast one of a data related to the input frame to produce bitstream withraster scan order information and displacement information for producingcompressed video bitstream, at decoding time, decoding at least one ofthe encoded bitstream with raster scan order information anddisplacement information for producing compressed video bitstream.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 is an embodiment of a frame coding in horizontal raster scanorder;

FIG. 2 is an embodiment of a frame coding in vertical raster scan order;

FIG. 3 is an embodiment of a vertical sliding window scheme forhorizontal raster coding order;

FIG. 4 is an embodiment of a horizontal sliding window scheme forvertical raster coding order;

FIG. 5 is an embodiment depicting horizontal bottom-right to left;

FIG. 6 is an embodiment of a vertical right-bottom to top;

FIG. 7 is an embodiment of a portable device with video camera andmotion sensor;

FIG. 8 is an embodiment of a video stabilizer in accordance with theprior art; and

FIG. 9 is an embodiment of an improved video stabilizer.

DETAILED DESCRIPTION

The proposed method and apparatus add a vertical raster scan order tothe HEVC standard and adaptively select the best coding scan orderbetween the horizontal and vertical raster scan orders based on anycriteria, such as, size of horizontal/vertical motion, codingefficiency, on-chip memory saving and etc.

The vertical raster scan is effective for the frames which have lots ofvertical discontinuities. FIG. 1 is an embodiment of a frame coding inhorizontal raster scan order. FIG. 2 is an embodiment of a frame codingin vertical raster scan order. In FIG. 1 and FIG. 2, the frame withvertical discontinuities with horizontal and vertical raster scan ordersare shown, respectively. When we apply the horizontal raster scan order,as shown in FIG. 1, more bits are used to encode micro block mode. Thisis due to incorrect mode prediction caused by discontinuity. However,the number of bits is reduced when we apply the vertical raster scanorder, as shown in FIG. 2. Hence, the coding gain will be moresignificant for intra frames. The result indicates that the verticalraster scan can reduce bitrates up to 4% for some sequences, evenwithout adaptive horizontal/vertical raster scan order decision. Thus,adaptive decision will result in even higher bit-rate saving.

FIG. 3 is an embodiment of a vertical sliding window scheme forhorizontal raster coding order. If a video contains large verticalmotion, the vertical sliding window scheme may not cover the largevertical motion. In this scenario, inter mode may not be used becauserequired reference area is not available. This eventually causesencoding efficiency degradation. However, this problem could be solvedby employing a horizontal sliding window scheme with vertical rastercoding order, as illustrated in FIG. 4. FIG. 4 is an embodiment of ahorizontal sliding window scheme for vertical raster coding order.Utilizing adaptively choosing raster scan order improves the codingefficiency of inter frames at the same on-chip memory requirement. Onthe other hand, the required on-chip memory size without coding gainloss is reduced.

For horizontal and vertical search range srX*srY, on-chip memory size(byte) for sliding window for 8-bit luma can be calculated as follows:

MemSizeHorOrder=picWidth*(2*srY+N),

MemSizeVertOrder=picHeight*(2*srX+N).

MemSizeHorOrder and MemSizeVertOrder are on-chip memory sizes forvertical sliding window (horizontal raster coding order) and horizontalsliding window (vertical raster coding order), respectively. N*N is thelargest coding unit, and picWidth and picHeight are the horizontal andvertical size of the picture. Based on the equations, Table 1 lists thesearch ranges for different available on-chip memory sizes for 4k×2k(3840×2160) videos. For a given on-chip memory size, srY is alwayslimited in vertical sliding window, which potentially causes encodingefficiency degradation for large vertical motion videos as illustratedin FIG. 3. By introducing vertical coding order with horizontal slidingwindow, we can remove the limitation on srY and avoid the degradationfor large vertical motion videos.

TABLE 1 Available on-chip memory sizes vs. search ranges 500 KBytes 750Kbytes 1,000 KBytes srX srY srX srY srX srY Vertical unlim- +/−33 unlim-+/−65 Unlim- +/−98 sliding ited ited ited window Horizontal +/−83unlimi- +/−141 Unlim- +/−199 unlim- sliding ted ited ited window

The idea of vertical raster scan can be realized with frame rotation.Hence, similar effect may be seen by rotating input frames withhorizontal raster scan. This information can be simply added as SEI(Supplemental Enhancement information) or VUI (Video UsabilityInformation) in bitstreams. Moreover, we can extend this idea to anyarbitrary raster scan order. We can apply one of 8 different raster canorders for each frames; horizontal top-left to right, as shown in FIG.1, horizontal top-right to left, horizontal bottom-left to right,horizontal bottom-right to left, as shown in FIG. 5, vertical left-topto bottom FIG. 2, vertical left-bottom to bottom, vertical right-top tobottom and vertical right-bottom to top FIG. 6. It also can be realizedby modifying input frames, i.e., e.g. rotating and/or flipping using SEIor VUI.

FIG. 7 is an embodiment of a portable device with video camera andmotion sensor. The motion sensor consists one of the following sensorsor a combination of them: accelerometer, gyroscope, magnetosensor etc.The motion sensor provided the camera orientation or displacementinformation. The displacement can be specified in terms of a rotationmatrix, quarternion, euler angle etc.

FIG. 8 and FIG. 9 show a video stabilization where camera displacementinformation can be used. Video stabilization compensates for jitter invideo due to changing camera position in order to provide a steadyvideo. FIG. 8 is an embodiment of a video stabilizer in accordance withthe prior art. In FIG. 8, the video stabilization is carried out indecoder. The camera displacement parameters are deduced from decodedframes and are used to translates/rotate/warp the decoded frame togenerate stabilized video frames. Calculation of camera displacementparameters from decoded frames is very computationally intensive andprone to incorrect estimation.

On the other hand, FIG. 9 is an embodiment of an improved videostabilizer. The camera displacement information is calculated in thetransmitter by using motion sensors. The displacement information isencapsulated in SEI/VUI and transmitted inside compressed bitstreams toreceiver. The decoder at the receiver decodes the compressed bitstreamwith camera displacement SEI/VUI and generates decoded frames anddisplacement information which is then passed on to the videostabilization module which carries out video stabilization.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

What is claimed is:
 1. A method of a digital processor for frame codingin adaptive raster scan order, comprising: decoding at least one of theencoded bitstream with raster scan order information and displacementinformation for producing compressed video bitstream.
 2. The method ofclaim 1, wherein at least one of the data and the displacementinformation is embedded in at least one of a picture header, a SEI and aVUI message.
 3. An apparatus for frame coding, comprising: a decoder todecode the encoded bitstream for decoding frames with at least one ofraster scan order information and displacement information for producingcompressed video bitstream.
 4. The apparatus of claim 3 furthercomprising a stabilizer for stabilizing the decoded frames withdisplacement information.
 5. A non-transitory computer readable mediumwith computer readable instructions, when executed perform a method forframe coding in arbitrary raster scan order, comprising: decoding atleast one of the encoded bitstream with raster scan order informationand displacement information for producing compressed video bitstream.6. The non-transitory computer readable medium of claim 5, wherein atleast one of the data and the displacement information is embedded in atleast one of a picture header, a SEI and a VUI message.